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Executive Summary 


At the first anniversary of school closures due to COVID-19, nearly half of the K-12 students in the 
U.S. were attending schools that were either fully remote or offering hybrid instruction, with more 
than 70 percent of California students attending schools remotely. For this reason, continued 
efforts to unpack the effects of COVID-19 on student outcomes are especially important for 
California students, who may be experiencing larger-than-average effects of continued school 
closures relative to the nation overall. 


In this report, we used data from multiple interim assessments to examine how the rate of 
student learning from fall 2019 through winter 2020-21 differs from that of student learning 
before COVID-19. Specifically, we assessed the degree to which approximately 100,000 students 
across 19 local education agencies (LEAs) in California experienced slower academic growth 
compared to previous school years (i.e., a learning lag) by the time they completed winter 2021 
interim assessments (NWEA MAP Growth, Renaissance Learning Star, and Curriculum Associates 
i-Ready) in Grades 4-8. To understand equity gaps in the degree to which students have 
experienced lost instructional opportunities, we disaggregated these results for students who 
were economically disadvantaged, students who were English learners, students with disabilities, 
students of different racial/ethnic backgrounds, students with low prior achievement, and 
students who were experiencing homelessness. 


We opted to use the term learning lag rather than learning loss in order to underscore that a lag 
in learning can occur relative to expected progress, even as students continue to learn and 

gain new knowledge and skills, and also that learning that has been delayed during the pandemic 
can be recouped through deliberate intervention. Our results show that by the time students 
completed winter interim assessments in the 2020-21 school year, they had experienced a 
learning lag of approximately 2.6 months in English language arts (ELA) and 2.5 months in math. 
We further found that students who were economically disadvantaged, English learners, and 
Latinx experienced greater learning lag than students who were not in these groups. We position 
these findings in the context of other recent studies that have estimated COVID-19 impacts 

on student learning, discuss caveats of using interim assessments (including those administered 
remotely), and highlight the importance of examining differences in the effects for students 

in different groups. These findings can be useful for guiding decision-making and resource 
allocation at the state and local levels. 
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COVID-19 Impacts on Student Learning: 
Evidence from Interim Assessments in California 


More than one year has passed since schools across the US. closed their doors in 
response to the global COVID-19 pandemic. As of March 22, 2021, roughly half of students in the 
U.S. were still attending schools that were either fully remote or offering hybrid instruction, with 
more than 70 percent of California students attending virtual-only schools at that time (Burbio 
School and Community Events Data Platform, 2021). Studies have also established concerning 
differences regarding in-person learning opportunities that continue to disproportionately affect 
students who are economically disadvantaged and students of color. Last spring, 22 percent of 
surveyed parents in California reported that their children had no live contact with their teachers 
(via phone or internet), with an even higher share of parents of low-income (28 percent), Latinx 
(26 percent), and Black (37 percent) students reporting no live contact (Gao et al., 2020). And, as 
of April 30, 2021, the percentage of students attending school in person and full time was three 
times higher for schools serving the least number of low-income families than those serving the 
most low-income families (Willis & Fensterwald, 2021). 


Since the start of school closures, multiple studies have emerged that predicted the 
potential impact of COVID-related school closures on students’ academic performance 
(e.g., Kuhfeld, Soland, et al., 2020; The Center for Research on Student Outcomes, 2020) and that 
assessed the magnitude of the impact on student learning given available data (e.g., Curriculum 
Associates, 2020, 2021; Domingue et al., 2021; Dorn et al., 2020; Kogan & Lavertu, 2021; Kuhfeld, 
Tarasawa, et al., 2020; Renaissance Learning, 2020). Though differing in the assessments and 
methodology used, these studies converged on a similar conclusion: students showed slower 
academic progress in the 2019-20 school year than would have been expected in the absence 
of COVID-related school closures. These studies also generally found more severe impacts on 
math achievement than on ELA or reading. 


Given the absence of statewide summative assessments in spring 2020, these studies have 
relied on the use of interim assessments to examine differences in learning patterns between 
pre-COVID-19 school years and COVID-affected school years. It is important to note that interim 
assessments are not equivalent to statewide summative assessments, which are designed to 
measure student proficiency and growth for informing high-stakes education policy decisions at 
local and state levels, among other purposes (O'Keefe & Lewis, 2019). Interim assessments are 
often shorter than statewide summative assessments, and therefore do not reflect all the learning 
standards that are expected to be taught in a given school year. Additional work is needed to 
systematically evaluate differences in various interim assessments, as well as differences between 
statewide summative and interim assessments (Dadey & Gong, 2017). Nevertheless, interim 
assessments are the best source of large-scale data about student learning available during 
the pandemic. They are by no means the only source of information about how students are 
progressing. Indeed, other recent research aims to leverage alternative sources of data, such as 
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student well-being survey responses, to gain a complete picture of the impact of COVID-19 on 
students (e.g., Wang et al., 2021). 


In addition, interim assessments in California have been administered almost exclusively 
at home as students continued distance learning either entirely or partially. Recent analyses from 
NWEA and Renaissance found that assessments administered remotely versus in person in fall 
2020 showed similar levels of reliability (Kuhfeld, Lewis, et al, 2020; Renaissance Learning, 2021); 
however, the Renaissance report did find that, in the winter, scores for early grades were higher for 
remote testers than for in-school testers. Our results from learning lag analyses in South Carolina, 
where we had a student-level indicator of where most students completed the assessment, 
indicate that the students who completed winter interim assessments remotely experienced 
greater learning lag than those who completed the assessments in person (Meyer et al., 2021). 
Because we do not have any data at the student or school level in California about whether 
students were tested remotely or in person, we cannot tease apart whether and how learning lag 
estimates might differ for students in different testing or learning environments. One encouraging 
finding is that the correlations between students’ fall 2019 and winter 2021 interim assessment 
scores, while lower than similar fall-to-winter correlations in previous years (not unexpected given 
the influence of COVID-19), were not lower by a dramatic degree. A representative example 
is the case of eighth graders’ ELA scores in our MAP sample: although the correlation between 
fall 2018 and winter 2020 scores was 0.85, it was 0.79 between fall 2019 and winter 2021. 


Among the studies leveraging interim assessments to compare student learning 
pre-COVID-19 to during the pandemic, few studies to date have used assessments that are more 
recent than the beginning of the 2020-21 school year (Curriculum Associates, 2021; Renaissance 
Learning, 2021). In addition, one challenge that has emerged from the many studies measuring 
COVID-19 impacts on student learning is the difficulty of drawing direct comparisons across 
multiple assessments because each study has used a unique methodology and has focused on 
one assessment. 


To address these gaps, we are reporting our recent research results to contribute to 
what we Know about how much students’ learning is lagging (or accelerating) relative to 
the rate we would expect, based on students’ growth in pre-COVID-19 school years and on 
interim assessment data from winter 2021.1 We include results from three different interim 
assessments—the NWEA MAP Growth, Renaissance Learning Star, and Curriculum Associates 
i-Ready—administered across 19 local education agencies (LEAs) in California that are part of the 
CORE Data Collaborative? In doing so, we aim to contribute to and expand upon the existing 
research documenting the magnitude of COVID-19 impacts on learning across assessment types, 


1 We also include results from fall 2020 interim assessment data, reported in the Appendix. 
? The CORE Data Collaborative is a consortium of more than 200 LEAs across California that focuses on school and student 
improvement through highly productive and meaningful partnerships between member school districts. 
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grades, subjects, and student characteristics. We disaggregated our results by multiple student 
characteristics in order to answer questions about differential impacts on particular student groups. 


We opted to use the terms learning change, learning lag, and learning acceleration rather 
than the more commonly referenced learning loss. We used learning change to describe our 
overall measures, learning lag to describe when those measures indicate students’ growth was 
slower during the COVID-affected years than pre-COVID-19 years, and learning acceleration to 
describe when those measures indicated students’ growth was faster during the COVID-affected 
years than the pre-COVID-19 years. We selected these terms in order to emphasize that a lag 
in learning can occur relative to expected progress—even as students continue to learn and 
gain new knowledge and skills—and that learning that has been delayed during the pandemic 
can be recouped through deliberate intervention. Although the results in this report are not 
causal estimates of schools’ (or educators’) impacts on student learning, they nevertheless 
can be considered empirical estimates of students’ academic progress during the pandemic, 
as influenced by school systems, educators, families, and communities. These estimates can 
therefore help identify which grades and which student subgroups were most affected by the 
COVID-19 disruption to schooling so that LEAs and state education agencies can provide the 
necessary resources to help students and educators emerge from this crisis. 


In the following sections, we outline our methods, including how we defined the sample 
of students and LEAs in these analyses, as well as the models we used to estimate learning 
change as of the winter of the 2020-21 school year. We then summarize the results for overall 
learning change and different student subgroups. We conclude with a discussion of the 
results, an acknowledgement of the limitations of our analysis, and a brief preview of future work 
in this area. 


Methods 


In this report, we summarize the results of “fall-to-winter” models that examined student 
growth from fall 2019 to winter 2021 and how that growth compared to average fall-to-next- 
winter growth in the last two years. This approach enabled us to capture the full impact of the 
pandemic to date. In the Appendix, we also report results that examined student growth from fall 
2019 to fall 2020, as reported in a recent PACE commentary. We do not discuss the fall-to-fall 
results in detail in this report, but we do provide the results in the Appendix for interested readers. 
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Sample 


In our analysis, we included approximately 100,000 unique students within the 19 California 
LEAs who are members of the CORE Data Collaborative. Almost all of these LEAs were offering 
distance-only learning, with a few smaller districts offering hybrid instruction as an option at some 
point between fall 2020 and winter 2021. In order to be included in the sample used to produce the 
learning change measures, an LEA needed to provide sufficient interim assessment data to measure 
academic growth for a given grade, subject, and assessment type in the most recent growth year 
(i.e., fall 2019 pretest and winter 2021 posttest) and in at least one of the two prior growth years 
(i.e., fall 2017 pretest and winter 2019 posttest, and/or fall 2018 pretest and winter 2020 posttest). 
Figure 1 provides a visual of these data requirements for LEAs to be included in each model. 


Figure 1. Local Education Agency Data Requirements for Learning Change Model 


At Least One Required 


COVID-19 
Fall Fall Winter Fall Winter Fall Winter 
2017 2018 2019 2019 2020 2020 2021 


Table 1 displays the number of students included in our sample in each posttest grade? 
and for each assessment type and growth year. We did not measure learning lag in math using 
the i-Ready assessment due to data only being available for a small number of districts. In 
addition, we only had one historic growth year (fall 2018-winter 2020) of i-Ready assessment 
data in ELA. 


In prior work examining fall-to-fall growth (Pier et al., 2021), we also reported results for high school grades. In this report, 
we restricted our analysis to Grades 3-8 due to smaller sample sizes in high school. These smaller sample sizes led to less precise 
learning lag estimates and more uncertainty around appropriate interpretation of results. 
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Table 1. Number of Students in Each Posttest Grade by Assessment, Subject 
(English Language Arts—ELA—and Math), and Growth Year 


Panel A: MAP 
Fall 2017 to Winter 2019 _— Fall 2018 to Winter 2020 _Fall 2019 to Winter 2021 

ELA Math ELA Math ELA Math 
Grade 4 6,690 6,704 6,676 6,711 4,957 4981 
Grade 5 7,160 7,217 6,677 6,740 5,214 5,198 
Grade 6 6,705 6,501 5,342 5,007 4133 3,925 
Grade 7 7,250 7,393 4,998 5,340 4,666 471 
Grade 8 6,295 6,390 5,146 5,135 4,081 4,206 
Totaln 34,100 34,205 28,839 28,933 23,051 23,021 


Panel B: Star 


Fall 2017 to Winter 2019 Fall 2018 to Winter 2020 Fall 2019 to Winter 2021 


ELA Math ELA ET) ELA Math 
Grade 4 4,131 915 4,117 980 3,521 946 
Grade 5 4,114 939 3,897 941 3,364 983 
Grade 6 2,966 478 3,687 394 3,248 828 
Grade 7 2,414 357 2,878 V2 3,100 457 
Grade 8 2,661 274 2,929 406 3,121 337 
Totaln 16,286 2,963 17,508 2,893 16,354 3,551 


Panel C: i-Ready 


Fall 2018 to Winter 2020 Fall 2019 to Winter 2021 


ELA Math ELA Math 
Grade 4 1,318 n/a 2,015 n/a 
Grade 5 1,986 n/a 2,149 n/a 
Grade 6 2,147 n/a 2,209 n/a 
Grade 7 2,915 n/a 2525 n/a 
Grade 8 3,512 n/a 2,761 n/a 
Total n 11,878 0 11,657 0 
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Table 2 reports our sample across growth years by student subgroup, combined across 
assessments to aid interpretability, though our models were all run separately by assessment type 
(see Table Al in the Appendix for these sample numbers separated by assessment type). Note 
that we defined students who were economically disadvantaged and English learners to match 
the way the California Department of Education defines these groups (California Department of 
Education, 2019, p. 39). Comparing the sample across growth years allowed us to gain a sense of 
whether students in a particular subgroup disproportionately dropped out of our sample during 
COVID-19 compared to before COVID-19. 


Table 2. Percent (and Number) of Students in Each Student Subgroup by Year versus California 
State Student Population 


2019-20 Fall 2017 Fall 2018 Fall 2019 
California State to Winter 2019 to Winter 2020 to Winter 2021 
xe) oLULE-La(ola) 
ELA Math ELA Math Math 
Native American and <1 <i <i <i <1 <i <1 
Pacific Islander (30,282) (166) (116) (243) (142) (229) (128) 
Asian American . : a . ? . 
(575,067) (1,766) (762) (2,723) (600) (2,620) (588) 
Black 5 6 9) 5 4 4 3 
(324,496) (2,897) (1,684) (2,954) (1,199) (2,231) (782) 
iatink 55 73 78 67 79 64 75 
(3,381,198) (36,955) (28,900) (38,935) (25,014) (32,800) (20,043) 
White 22 15 12 18 12 19 1S 
(1,381,737) (6,525) (4,579) (10,524) (3,899) (9,829) (3,564) 
Fhglizhy Learners 19 27 27 22 24 22 25 
9 (1,148,024) (13,381) (10,162) (12,841) (7,743) (11,232) (6,557) 
Students with 12 13 14 14 15 9 il 
Disabilities (721,198) (6,788) (5,382) (7,983) (4,714) (4,777) (2,834) 
Economically 61 79 81 70 80 68 78 
Disadvantaged (3,741,755) (39,743) (30,232) (40,472) (25,542) (34,580) (20,858) 


Table 2 indicates that when comparing the pre-COVID-19 and COVID-year samples 


combined across assessments, there are proportionally fewer Black and Latinx students and 


more White students (in ELA only), though the decreases appear to be gradual across the years 


rather than dropping of 
of English learners and 
winter 2020 (pre-COV| 


ff in the COVID-year sample (i.e., fall 2019-winter 2021). The proportion 
economically disadvantaged students is smaller in both the fall 2018- 
D-19) sample and the fall 2019-—winter 2021 (COVID-affected) sample 


compared to the fall 2017-winter 2019 (pre-COVID-19) sample. Together, these changes imply 
that, although the demographics of our sample did shift slightly over time, this does not appear 
to be due to a dramatic change in the students who have been assessed in the COVID-affected 
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years relative to the pre-COVID-19 years. The exception is students with disabilities, who are 
underrepresented in the COVID-affected growth year relative to the pre-COVID-19 growth years. 


In addition, Table 2 includes the percent (and number) of students in each demographic 
group in the 2019-20 California state student population’ in order to show how representative 
our sample is of the state population. Our sample has a lower proportion of Asian American 
students across all years, a lower proportion of Black students (in math only) that becomes 
ower over time, and a lower proportion of White students (which becomes more similar to 
the state population in ELA over time). Our sample has a higher proportion of Latinx students, 
English learners, and economically disadvantaged students than the state population. Our sample 
of students with disabilities is similar to the state population in pre-COVID-19 years but lower 
than the state population in COVID-affected years. In summary, our sample generally has an 
overrepresentation of Latinx, English learner, and economically disadvantaged students compared 
to the state population as well as an underrepresentation of Asian American, Black, and White 
students compared to the state population; we further observed fewer students with disabilities 
assessed in the COV|D-affected years compared to the state population. 


Overall Approach 


We used four years of fall and winter interim assessment data (from fall 2017-18 through 
winter 2020-21) to compute three years of academic growth from fall to the following winter 
(fall 2017—-winter 2019, fall 2018-winter 2020, fall 2019-winter 2021). For a given period (e.g., fall 
2017-winter 2019), students needed to have assessment data for both the pretest (e.g., fall 2017) 
and the posttest (e.g., winter 2019). For simplicity, we labeled each winter with the latter 
number of the school year (e.g., winter 2018-19 is referred to as winter 2019). 


In most cases, we defined a fall assessment as One attempted between August 1 and 
ovember 25 and a winter assessment as one attempted between November 26° and the last day 
of February. We made an exception to this for the i-Ready assessment because using these dates 
for the fall i-Ready assessment led to intervals of time between posttest and pretest assessments 
that were substantially different in the COVID-affected and pre-COVID-19 years. This difference 
does not reflect a problem in the assessment or its administration; it was a timing phenomenon 
that happened to confound this particular analysis. For the i-Ready, we defined a fall assessment as 
one attempted between August 1 and October 31. We kept the winter assessment defined as one 
attempted from November 26 to the last day of February. 


* For race/ethnicity data, see https://dq.cde.ca.gov/dataquest/dqcensus/EnrEthGrd.aspx?cds=O00&agglevel=state&year=2019-20. 
For English learner, students with disabilities, and economically disadvantaged data, see https://dq.cde.ca.gov/dataquest/dqcensus/ 
EnrCharterSub.aspx?cds=OO0Sagglevel=stateSyear=2019-20. 

° We chose November 26 as the cutoff date by identifying the earliest date for the first Monday after Thanksgiving between 2017 
and 2020, as winter testing typically begins after the Thanksgiving break. 
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We first computed growth models separately for each grade (Grades 4-8, where the 
grade corresponds to the posttest grade but represents growth from the prior fall until the 
following winter), subject (ELA and math), and assessment type (MAP, Star, and i-Ready) using 
data pooled over all available years. The growth models calculated a predicted score for a 
student's posttest based on that student's pretest score and demographic information (English 
learner, disability, economic disadvantage, and homelessness status). We then measured growth 
as the difference between the student's predicted posttest score and the student's actual 
posttest score. Those growth measures were averaged across students in the same year, grade, 
subject, assessment type, and LEA to produce an overall estimate of growth. The sample 
over which this growth model was estimated included LEAs for which we had assessment 
data in any growth year—pre-COVID-19 or COVID-affected. This was a larger sample than 
that described above and included some LEAs that were not included in the learning change 
measures themselves due to not having growth measures both in a pre-COVID-19 growth year 
and in the COVID-affected growth year. 


We then calculated the learning change, which is the difference between growth in the 
most recent year and the average growth In the two prior years. Specifically, learning change 
compares (i) growth from fall 2019 to winter 2021 and (ii) the average growth from fall 2017 to 
winter 2019 and fall 2018 to winter 2020. Note that learning change can only be measured in 
LEAs where growth is measured in both a pre-COVID-19 growth year and in the COV|D-affected 
growth year; these LEAs make up the sample described in the previous section. 


Learning change is always calculated by comparing growth from year to year for students 
in the same grade, subject, assessment, and LEA; as a result, these measures compare different 
cohorts of students. In other words, the learning change measure compares fall-to-winter 
growth between the current cohort of students in a grade and subject to that of previous cohorts 
of students in that same grade and subject. Thus, our model is structurally similar to a quasi- 
evaluation model in which the students in the COV|D-affected period represent a treatment group 
and the students in the pre-COVID-19 period represent a control group. To control for potential 
differences across the treatment and control groups caused by changes in the demographic 
composition of the samples and differences in test participation, we used a multilevel growth 
model that controls for prior achievement, student demographic characteristics, and district of 
enrollment. We averaged learning change measures across LEAs to compute learning change 
across Our sample in California. 


In order to visualize and summarize results across different assessments and to help 
make the magnitude of learning change observed more intuitive for readers, we converted our 
results from scale scores to a months of learning metric. To do this, we used a transformation 
based either on a typical amount of growth from one year to the next or on the amount of 
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variation in attainment within a given year.® Although we appreciate the interpretability that this 
metric provides, we also caution readers that this conversion is an approximation. 


Model Specification 
The fall-to-winter growth and learning change model is given by the following formula: 
Yije = Vije-1A + Xijt-1B + Ae + ize (1) 
where: 


e student/ is in LEA/ in year ¢; 

° ijt yis student /’s score in the winter of year t; 

* Yijt-1 IS student /’s score in the fall of year t-1,; 

¢ Xije-1 1S a vector of student-level variables (disability, English learner, economic 
disadvantage, homelessness status, and number of days between assessments); and 

° Qj are LEA-year fixed effects. 


We estimated Equation 1 over a multi-LEA and multiyear sample of students for a single 
grade, subject, and assessment type. This sample pools observations over three years so that 
some observations measured growth from fall 2019 to winter 2021, others measured growth 
from fall 2018 to winter 2020, and yet others measured growth from fall 2017 to winter 2019. 
Note that observations for different years will be for different cohorts of students. For example, 
the sample for the MAP assessment for Grade 4 will pool growth for three cohorts of students 
progressing from third to fourth grade:’ those students progressing from Grade 3 to Grade 4 
between the 2019-20 and 2020-21 school years; those progressing from Grade 3 to Grade 4 
between the 2018-19 and 2019-20 school years; and those making the same progression 
between the 2017-18 and 2018-19 school years. 


© For the MAP, we measured a month of growth by dividing fall-to-spring growth norms by 7.5 for a given grade. For the i-Ready, 

we measured a month of growth by computing the difference between the 50th percentile achievement norms for fall and spring 
and dividing the result by 7.5. For the Star, we used one of two approaches. In some subjects and grades (reading Grades 4-7, math 
Grades 4-5), we measured a month of growth by computing the difference between the achievement norm at the beginning of the 
year and the achievement norm in the ninth month of the year, and by dividing that difference by 9. In other subjects and grades 
(reading Grade 8, math Grades 6-8), we measured a month of growth by computing the difference between the achievement 
norms at the 31st and 69th percentiles (corresponding to a difference of one standard deviation around the median, assuming a 
normal distribution), multiplying that difference by a 10-month longitudinal standardized growth norm presented in Table 1 of Lee et 
al. (2011), and dividing the result by 10. We chose one approach over the other by whichever one led to the greater number of scale 
score points per month of learning. (Applying a similar rule to the MAP or the i-Ready always led to using an approach similar to the 
first approach, based on average growth rather than standard deviation of attainment.) In all cases, the conversions were based on 
growth or achievement norms for the year of the pretest (e.g., Grade 3 when measuring growth from Grade 3 to Grade 4). 

7|n practice, the sample also included a very small number of students who repeated a grade or skipped a grade. We defined these 
samples according to the grade of the outcome assessment. We allowed into the sample the small number of students who were 
in a grade different than the previous one in the previous fall. For example, a student who was in Grade 4 in fall 2019 and Grade 6 in 
winter 2021 would have been included in the Grade 6 analytic sample. 
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The model in Equation 1 was estimated using errors-in-variables regression (Fuller, 1987) 
to account for measurement error in the pretest Yije-1. Estimating this regression yields slope- 
coefficient estimates J and B and LEA-year fixed-effects estimates @r. We estimated learning 
change for an individual LEA / by taking the LEA's fixed-effect estimate for the COVID-affected 
year (2020) and subtracting from it a weighted average of the same LEA’s fixed-effect estimates 
for the non-COVID-19 years (2018 and 2019): 


> —s nj19 5 nNj18 a 
€) = G20 — ( Qji9 + —H8— G19] (2) 


NjistNji9 NjigtNji9 
where nj is the number of students in the sample associated with LEA / in year t. 


We produced a precision-weighted aggregate learning change measure # for a given 
combination of assessment type, subject, and grade by averaging across LEA-level learning 
change measures 2, using the inverse of the squared standard error of 2; as a weight. We 
expressed this mathematically in Equation 3: 


where 6gcj) is the estimated standard error of 2. This approach to weighting effectively applies 
more weight to the LEAs with more students, for whom estimates of learning change are more 
precise. 


By estimating learning change one LEA at a time and then averaging learning change 
across LEAs to produce an overall learning change measure, we avoided problems from changes 
in the composition of students by LEA in our sample. In addition, controlling for Yije-1 and 
Xije-1 Snould help control for differences across years in student prior attainment and student 
demographics in our sample. 


tis useful to note that the set of control variables Xjj¢_-4 includes the number of days 
between the fall and winter assessments used to measure growth. This is especially relevant on 
the MAP assessment, where the number of days between assessments was often one month 
onger in 2020-21 than in the pre-COVID-19 comparison years. It is also relevant to note that, 
in the MAP sample, there were substantial differences between 2020-21 and the pre-COVID-19 
years in the proportion of students taking the assessment before and after the winter break. For 
the MAP, about four fifths of students in pre-COVID-19 years took the winter assessment before 
the break, while nearly all students in 2020-21 took the winter assessment after the break. 

To check the extent to which this could affect the results, we re-estimated the model using an 
alternative specification in which we included the number of days between assessments and 
an indicator variable for whether the winter assessments were administered in January or later. 
The finding of learning lag was robust to this alternative specification, although we did find that 
learning lag was on average about one MAP scale score point smaller in math, corresponding 
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to about three quarters of a month of learning. The Star and i-Ready assessments were more 
balanced, and results were largely unaffected by this alternative specification; in both the 
pre-COVID-19 years and in 2020-21, most students took the winter assessment after the break 
using the Star, while about two thirds of students took the winter assessment before the break 
using the |-Ready. 


Measuring Learning Change by Student Subgroup. We were interested in the extent 
of learning change overall and the extent of learning change for specific student subgroups. 
For example, we may be interested in learning change among English learners. To produce 
subgroup-specific learning change measures, we began by producing a student growth measure 
equal to 


Gije = Vije — Vije-14 — Xije-1B (4) 


The measure @jjt is an estimate of student growth from fall to winter that takes into account 
the extent to which students with higher or lower fall achievement Yije-1 Or different characteristics 
Xijt-1 tend to grow more or less in general. To obtain estimates of subgroup-specific growth and 
learning change, we computed the means of student growth for each subgroup for each LEA in 
each year. This calculation produced estimates of growth for each subgroup similar to the fixed 
effect estimates @. As above, we produced LEA-and-subgroup-specific learning change 2; and 
overall subgroup-specific learning change @ using Equations 2 and 3. 


These subgroup-specific learning change measures compared the growth of students in 
that subgroup in the year affected by COVID-19 to the growth of students in that subgroup In 
years unaffected by COVID-19. For example, a learning change measure produced for English 
learners in the way described above measures the difference between the growth of English 
learners in the COVID-disrupted year and the growth of English learners in pre-COVID-19 years. 
It does not measure whether student growth is lower for a specific subgroup relative to other 
subgroups. Rather, it measures whether student growth for that subgroup is lower than what it 
had been for the same subgroup in the past. 
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Results 


Our fall-to-winter models answered the following question: How did students progress 
academically from fall 2019 to winter 2021, compared to how they progressed over the same 
time frame in prior school years? In other words, the models measured the degree to which 
students grew faster or slower in 2019-20 and into the 2020-21 school year, a period in which 
COVID-19 emerged and schools provided instruction either entirely or partially via distance 
learning, compared to an equivalent time period in pre-COVID-19 years. In the results below, 
we identify students by their posttest grade. However, it is important to note that the learning 
change estimates describe the growth that occurred during the entire school year in the pretest 
grade and at the beginning of the school year in the posttest grade; in other words, the learning 
change results for fourth graders describe their growth from the beginning of third grade until 
the winter of fourth grade. 


Figure 2 displays results across subjects, grades, and assessment types in units of months 
of learning (on the vertical axis). The horizontal axis displays the students’ grades during the 
2020-21 school year (i.e., the posttest grade). The vertical dotted black lines depict 95 percent 
confidence intervals. Dotted lines that cross zero indicate that the estimate is not statistically 
significantly different from zero, indicating we cannot conclusively determine If students’ 
academic growth from 2019-20 was faster or slower than growth in prior years. For example, 
in ELA, fourth-grade students experienced 3.3 months of learning lag on the MAP, 2.7 months of 
learning lag on the Star, and 0.7 months of learning lag on the i-Ready. All three ELA estimates 
for fourth graders are significantly different from zero, as indicated by the vertical dotted lines that 
are not crossing the horizontal axis. 


The results in Figure 2 show that we find evidence of learning lag across all three 
assessments in nearly all cases. We averaged the results across grades and assessments using 
an inverse-variance weighted mean that weights more heavily the results for grade-assessment 
combinations that are estimated more precisely. Using this average, we find that, compared to 
growth in prior years, students are experiencing a learning lag of approximately 2.6 months 
of learning in ELA and 2.5 months of learning in math; these are presented in the grey bars in 
Figure 2 labeled “All Grades.” However, it is important to note that there are substantial differences 
in the learning change results across subjects, assessment types, and grades. Specifically, learning 
change is not statistically significantly different from zero in either ELA or math among eighth 
graders when measured using the MAP. It is also not statistically significantly different from zero 
in ELA for seventh graders when measured using the i-Ready. In addition, results from the Star 
in math and results from the i-Ready in both math and ELA suggest that eighth-grade students 
experienced learning acceleration rather than lag. 
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Figure 2. Overall Learning Change 


ELA Math 


Learning Change in Months of Learning 


Grade Grade Grade Grade Grade All Grade Grade Grade Grade Grade All 
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Results for Student Subgroups. We extended the above analysis to examine how learning 
change differs for different student subgroups. We did this by examining whether a particular 
subgroup experienced significantly more or less learning change from fall to winter this year 
(relative to prior years) than their peers who were not in the subgroup. We included results for 
students who were economically disadvantaged, students who were English learners, students with 
disabilities, students of different racial/ethnic backgrounds, students with low prior achievement, 
and students experiencing homelessness. 


Figure 3 depicts the fall-to-winter results for students who were economically 
disadvantaged. On average, across grades and assessments, students who were economically 
disadvantaged experienced 3.2 months of learning lag in ELA and 2.8 months of learning lag 
in math, while students who were not economically disadvantaged experienced 1.1 months 
f learning lag in ELA and 1.7 months of learning lag in math. These averages are presented 
in Figure 3 in the panel on the far right, labeled “All Grades.” At a more granular level, students 
who were economically disadvantaged experienced greater learning lag in ELA than their non- 
disadvantaged peers in Grade 4 (except on the i-Ready), Grade 5, and Grade 6. In higher grades, 
the results were more mixed: Students who were economically disadvantaged experienced 
greater learning lag in ELA only in Grade 7 on the i-Ready and Grade 8 on the Star. In math, these 
students experienced significantly greater learning lag in Grades 4 and 5 on the MAP® Grades 5 
and 7 on the Star, and Grade 7 on the i-Ready. 


® Note that in math for eighth graders on the MAP, students who were economically disadvantaged experienced significantly less 
learning lag than students who were non-economically disadvantaged. 
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Figure 3. Fall-to-Winter Results: Economic Disadvantage 


Panel A: English Language Arts: Economically Disadvantaged versus Non-Economically 
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Panel B: Math: Economically Disadvantaged versus Non-Economically Disadvantaged 


Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All 
Grades 

A 

= 11 : 

£ ; 

c 9 4 

o 

oO 7 4 

eal 

i) 54 

a 

s 34 

Cc 

2 1 

£ -1 4 |. I. | 

o 

2 -3 : 

© : 

§ -s{-— 

Eg 75 

£ 

Co] =9'4 

Ss 

-11 


MAP Star iReady MAP Star iReady MAP Star iReady MAP Star iReady MAP Star iReady All tests 


MED W&Non-ED 


edpolicyinca.org 


VA PACE 


Figures A1—A5 in the Appendix display these results for English learners, students with 
disabilities, students of different racial/ethnic backgrounds, students at different levels of prior 
achievement, and students experiencing homelessness. Figure 4 summarizes learning loss by 
subgroup by depicting the average learning lag experienced by students in each subgroup across 
grades and assessment type. However, results vary by grade and assessment type (in terms of 
direction l[i.e., learning loss or gain], magnitude, and precision); therefore, we further called out 
which results were statistically significant in the paragraphs that follow. 


English learners (versus non-English learners) experienced, on average, 3.8 (versus 2.3) 
months of learning lag in ELA and 3.1 (versus 2.4) months of learning lag in math. Specifically, both 
the MAP and Star assessments indicated greater learning lag for English learners across grades. 


Students with disabilities (versus students without disabilities) experienced, on average, 
3.0 (versus 2.7) months of learning lag in ELA and 2.8 (versus 2.6) months of learning lag in math. 
However, the results by grade and assessment were mixed: the only statistically significant 
result was a greater learning lag for students with disabilities in eighth-grade math, as measured 
by the MAP. 


Latinx students experienced, on average, 3.4 months of learning lag in ELA and 2.8 months 
of learning lag in math—the largest magnitude of learning lag for any racial/ethnic subgroup in 
our sample—except Native American and Pacific Islander students in ELA (for which we have a 
small sample that is therefore imprecisely estimated). Specifically, we found greater learning lag 
for Latinx students in comparison to White students in ELA and math on all three assessments, 
with larger gaps in earlier grades. 


Students with low prior achievement (versus students who did not have low prior 
achievement) experienced, on average, 3.1 (versus 1.9) months of learning lag in ELA and 2.7 
(versus 2.4) months of learning lag in math. Most notably, we found more learning lag among 
low-achieving students in ELA on the MAP across grades.? 


Finally, students experiencing homelessness (versus students not experiencing 
homelessness) experienced, on average, 3.7 (versus 2.6) months of learning lag in ELA and 
34 (versus 2.5) months of learning lag in math. However, results are usually not statistically 
significant, likely due to the small number of students in our sample who were experiencing 
homelessness. 


° We define low prior achievement as students scoring in the bottom two performance categories on each assessment. The MAP 
performance categories are based on quintiles, such that students in the bottom two categories correspond to students in the 40th 
percentile or lower. The two bottom performance categories on the Star include “Did not meet standard” and “Nearly met standard,” 
and on the i-Ready include “Three or more grade levels below” and “Two grade levels below.” 
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Figure 4. Average Learning Lag (in Months of Learning) Across Grades and Assessments 
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To summarize, we observed more learning lag for students who are economically 
disadvantaged (except in upper grades in math, as measured by the MAP), English learners, and 
Latinx. We further observed evidence of learning lag across all assessments for students who 
ieving (except in ELA, as measured by Star, where results were more 
mixed). We found mixed results for students with disabilities, as they experienced more learning 
lag on the MAP (relative to students without disabilities) but less learning lag on the Star in ELA. 
Note that because we observed fewer students with disabilities in the COVID-affected years, it is 
possible we were less reliably estimating the learning lag for these students. These mixed results 
may be due to differences in the various assessments, differences in the LEAs that administer 
different assessments, or differences in how students actually experienced instruction between 


were previously low ach 


math and ELA. 


When examining the results of the fall-to-fall learning lag models provided in the Appendix, 
we find evidence that is suggestive of continued learning lag during the 2020-21 school year 
compared to the learning lag sustained by fall 2020. However, additional research is needed to 
quantify precisely how much additional learning lag was sustained. Nevertheless, our results 
here show the accumulated effect of school closures since the onset of the pandemic through 


winter 2021. 
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Discussion 


When assessing how much faster or slower students’ achievement has progressed during 
the prior school year and so far this school year (through winter 2020-21), we found evidence 
across assessment types, grades, and subjects that student learning is progressing more slowly 
than we would have expected in the absence of COVID-19, affecting the delivery of instruction. 
Our analyses indicate that students experienced 2.6 months of learning lag in ELA and 2.5 months 
of learning lag in math as of this winter, compared to expectations in a typical school year. 


However, eighth graders showed either comparable gains in achievement to prior years 
or showed evidence of some learning acceleration in math and ELA. Therefore, it is important 
to attend to the differences in results by grade and assessment type, even though we provide 
average effects for interpretability. 


When we disaggregated the results based on student characteristics, we observed that 
students who were economically disadvantaged, English learners, and Latinx experienced greater 
learning lag than did students who were not in those groups. In addition, we found that students 
who were previously low achieving experienced greater learning lag than students who were not 
previously low achieving. 


We wish to emphasize that we observed differences in the effects by grade, subject, and 
assessment type with all these results. For instance, as noted above, we did not observe learning 
lag consistently for eighth graders in math or ELA. We found mixed results for students with 
disabilities over both periods—with the MAP assessment suggesting a greater learning lag and 
the other assessments suggesting less learning lag in some grades and subjects. Seeing as we 
observed fewer students with disabilities assessed in 2019-20 and 2020-21 compared to in 
the pre-COVID-19 years, it is possible our results estimated learning lag for these students less 
reliably. For all the student subgroup results, there are assessment/grade/subject combinations 
where gaps are Statistically significant and others where they are not, and there are some 
combinations that snow the opposite result from the overall pattern. Our goal with this report is 
to highlight overall patterns that appear consistently across assessments and grades. We believe 
that, together, all of these findings provide stronger evidence of real patterns than If a pattern 
were consistently true for one assessment but not another. 


However, because we did not have a sample where students completed more than one 
assessment, It is impossible to disentangle whether differences in the pattern of results between 
assessments are due to differences in the assessments themselves (e.g., the degree of curriculum 
sensitivity), due to differences in the administration of the assessment (e.g., how remote 
assessments were conducted), or due to differences in the schools or LEAs where that particular 
assessment was used. 
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Related to this, the lack of available data at the student (or even school) level about 
whether students were tested or if they were learning fully remotely or in a hybrid environment 
made it impossible to sufficiently control for or differentiate by testing environment or learning 
environment. In addition, results may differ depending on whether students completed the winter 
interim assessment in a school building versus at home (which is correlated with but not equivalent 
to whether students received any in-person instruction). Although NWEA found that the MAP 
assessments administered remotely versus in person in fall 2020 showed similar properties 
(Kuhfeld, Lewis, et al., 2020), our results from learning lag analyses in South Carolina, where 
we did have a student-level indicator of where students completed the assessment for most 
students, are more consistent with findings from Renaissance Learning (2021)—specifically, that 
there may be notable differences in scores between remote and in-person testers in the winter. 
We found that students who completed the winter interim assessment remotely experienced 
greater learning lag than those who completed the assessment in person (Meyer et al., 2021). 
However, we did not have the data in South Carolina to tease out whether this was a function 
of taking the assessment remotely or learning remotely/at home. Nevertheless, we believe the 
California results presented here could differ depending on whether students completed 
the assessment remotely or in person—if we had the data to differentiate by this factor. Future 
research on COVID-19 impacts on learning should prioritize obtaining and analyzing any student- 
level data on assessment location and learning environment. 


Furthermore, it is also the case that we have a slightly smaller sample in the COVID- 
affected year than the pre-COVID-19 years. This opens up some possibility of bias if the smaller 
sample in the COVID-affected year is due to student assessment data being missing not at 
random. However, it is important to note a couple of things. First, there should only be bias if 
students are missing due to variables relevant to student achievement but not already controlled 
for in the growth model. For example, there should not be bias in our learning lag estimates 
if students are missing from the COVID-affected sample due to prior achievement, given that 
prior achievement is controlled for in the growth model. Second, the fact that the demographic 
composition of students stays mostly the same from year to year Is at least suggestive that the 
sample of students is broadly similar between the COVID-affected and pre-COVID-19 years. The 
only exception is students with disabilities, which is proportionally smaller in the COVID-affected 
year. Note, however, that disability is controlled for in the growth model and should not directly 
cause bias in the learning lag measures. 


Our findings generally align with those of other recent studies (e.g., Curriculum Associates, 
2020, 2021; Dorn et al., 2020; Kuhfeld, Tarasawa, et al., 2020; Renaissance Learning, 2020), with 
one exception: these studies generally found greater learning lag in math than in ELA, where we 
tended to see evidence of learning lag in both ELA and math. From the studies examining winter 
interim assessment data, Curriculum Associates (2021) reported learning lag in both subjects on 
the i-Ready, except for Grade 8 ELA; Renaissance Learning (2021) reported improvements in 
both subjects on the Star but less progress for students of color, students with disabilities, and 
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Our findings may differ from those of other studies due to differences in the sample, 
selected methodology, assessments used, and the metric we used to report the results (i.e., 
months of learning). Studies using nationwide samples (e.g., Curriculum Associates, 2020, 2021; 
Dorn et al., 2020; Kuhfeld, Tarasawa, et al., 2020; Renaissance Learning, 2020, 2021) may find 
different patterns than the ones we identified specifically for the California districts in our sample, 
as they may average across patterns within specific states when drawing conclusions that are 
generalizable across different states. Given the differences we noted in our results by assessment 
type, grade, and student subgroup, it seems beneficial to have both LEA- and state-specific results 
for informing policymaking and decision-making at those levels, as well as nationwide results 
to inform efforts at the federal level. 


Conclusion 


Students and educators across the country have experienced a loss of instructional 
opportunities due to COVID-related school closures. Yet, given that California accounted for 
nearly half of all students in the U.S. who were still attending schools with fully remote instruction 
in winter 2021 (Burbio School and Community Events Data Platform, 2021), it is imperative 
to understand the effect of these lost opportunities on California students’ learning. We found 
evidence that students in the California districts in our sample experienced a learning lag in 
Grades 4-8 math and ELA since the onset of the pandemic and through winter 2021. Crucially, 
students who are most often underserved by our educational system are experiencing more of 
this learning lag than their peers; students who are economically disadvantaged, English learners, 
and Latinx experience greater learning lag, which exacerbates existing opportunity gaps that 
were problematic and concerning prepandemic. 


The demographic composition of students in our sample did shift slightly over time, but 
changes do not appear to be due to a dramatic shift in the students assessed in COVID-affected 
years relative to pre-COVID-19 years. The exception is students with disabilities, who were 
underrepresented in the COVID-affected growth year relative to the pre-COVID-19 growth years. 
Thus, our estimates of learning lag for these students may be less reliable. 


There are several reasons to interpret the findings here with some caution. As previously 
mentioned, we did not have data available to distinguish whether students completed these 
assessments remotely or in person, nor did we have student-level data indicating whether 
students received any in-person instruction during the 2020-21 school year. Another caution is 
that our results here are presented in the converted months of learning scale for intuitiveness and 
interpretability across different assessments; despite the positive properties of this metric, It is 
also an approximation that should not be literally interpreted to correspond to a specific number 
of weeks or months of instruction. 
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Finally, we wish to emphasize that the learning change results presented here are not 
causal estimates of schools’ or LEAs’ impacts on student learning; rather, they reflect the 
contributions of schools and families, given that students have been learning partially or fully 
at home. There are far too many factors outside the control of educators—including students’ 
access to a reliable internet connection, students’ responsibilities for other children in their family, 
students’ access to an adult to help with their schoolwork when needed—that are relevant during 
COVID-19. Although student-level control variables (such as prior achievement) typically do a 
good job of controlling for these differences in pre-COVID-19 school years (Kane & Staiger, 2008; 
Chetty et al, 2014; Deming, 2014; Angrist et al., 2017), it is likely that during COVID-19, there were 
other factors not captured by these variables that would affect students’ learning. Therefore, these 
measures Capture the combined effects of schools and families—and thus could be thought of 
as community-level indicators of students’ learning while COVID-19 continues to limit instructional 
time and in-person instructional opportunities. Given this, these results can be useful for guiding 
decision-making and resource allocation at the state and LEA levels. More immediately, they can 
be used to inform decisions about participation in interventions designed to offset the negative 
effects of COVID-19 on learning (e.g., summer academic recovery programs). 


At Education Analytics, we are continuing to refine our approach for estimating and 
interpreting learning lag measures. This includes exploring additional modeling approaches 
(such as including school fixed effects), conducting further robustness checks of our results, 
and gathering more data from additional LEAs, additional assessment types, and additional 
assessment administrations (e.g., spring 2021). Looking forward, we aim to monitor the degree 
to which students experience learning recovery in response to the policies and interventions 
that districts and states are currently planning as we emerge from the pandemic. Our goal is to 
continue to deepen our understanding of how COVID-19 is affecting student learning within 
the states where we work and of its impact for the students who are most at risk of being 
inequitably affected, as well as to share those findings as they emerge to inform policymakers 
and educational stakeholders grappling with how to accelerate student learning this summer, 
next school year, and beyond. 
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Appendix 


Additional Fall-to-Winter Subgroup Results 


Table Al provides details on the sample by student subgroup for each growth year 
separately by assessment. Figures Al—A5 display the fall-to-winter results for English learners, 
students with disabilities, students of different racial/ethnic backgrounds, students with low prior 
achievement, and students experiencing homelessness. 


Table A1. Percent (and Number) of Students in Each Student Subgroup by Assessment and by 
Year versus California State Student Population 


Panel A: MAP 


2019-20 
California State 


Lxe) eLULE-La (ola) 


Fall 2017 
to Winter 2019 


Fall 2018 
to Winter 2020 


Fall 2019 
to Winter 2021 


ELA Math Math 


Native American and <1 <i <1. <i <a <i <1 
Pacific Islander (30,282) (89) (88) (110) (117) (97) (103) 
Asian American 9 2 2 2 2 2 2 
(575,067) (613) (616) (435) (445) (351) (351) 
Black 5 5 5 4 4 3 3 
(324,496) (1,544) (1,549) (1,068) (1,104) (614) (643) 
Latinx 55 81 82 84 83 85 85 
(3,381,198) (27,711) (27,926) (24,161) (24,000) (18,968) (18,982) 
White 22 9 9 8 8 8 8 
(1,381,737) (3,145) (3,042) (2,284) (2,448) (1,807) (1,737) 
English Learners 19 28 29 27 25 28 27 
(1,148,024) (9,706) (9,776) (7,795) (7,283) (6,223) (6,080) 
Students with 12 14 14 15 Hs) 11 11 
Disabilities (721,198) (4,884) (4,933) (4,377) (4,300) (2,442) (2,457) 
Economically 61 85 85 85 84 87 88 
Disadvantaged (3,741,755) (28,879) (29,052) (24,406) (24,373) (19,518) (19,563) 
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Panel B: Star 


2019-20 Fall 2017 Fall 2018 Fall 2019 
California State to Winter 2019 to Winter 2020 to Winter 2021 
exe) oLULE-La lola) 
ELA Math Math 
Native American and <1 <i <i <1 <1 <1 <i 
Pacific Islander (30,282) (77) (28) (86) (25) (86) (25) 
Asian American 9 7 5 7 5 7 7 
(575,067) (1,153) 146) (1,252) (155) (1,162) (237) 
Black 5 8 5 8 3 7 4 
(324,496) 1,353) 135) 1,355) (95) (1,141) 139) 
Latinx 55 57 33 53 oo 51 30 
(3,381,198) 9,244) 974) 9,331) (1,014) (8,379 (1,061) 
White 22 21 52 25 50 26 oul 
(1,381,737) 3,380) (1,537) 4,291) (1,451) 4,313 (1,817) 
English Learners 19 23 5 22 16 22 13 
(1,148,024) 3,675) (386) 3,836) (460) 3,607 477) 
Students with 12 12 5 12 14 8 11 
Disabilities (721,198) (1,904) (449) 2,015) (414) 1,367 (377) 
Economically 61 67 40 63 40 58 36 
Disadvantaged (3,741,755) (10,864) (1,180) (11,005) (1,169) 9490 (1,295) 


Panel C: i-Ready 


2019-20 


California State 


xe) oLULE-L (ola) 


Fall 2018 
to Winter 2020 


Fall 2019 
to Winter 2021 


Math Math 
Native American and <1 <i aia <1 os 
Pacific Islander (30,282) (47) (46) 
Asian American 9 9 Bie 10 ae 
(575,067) (1,036) (1,107) 
Black 5 5 aie 4 le 
(324,496) (551) (476) 
Latinx 55 46 ae 47 =e 
(3,381,198) (5,443) (5,453) 
White 22 33 ae 32 ais 
(1,381,737) (3,949) (3,709) 
English Learners 19 10 aes 12 aa 
(1,148,024) (1,210) (1,402) 
Students with 12 13 ie 8 ae 
Disabilities (721,198) (1,591) (968) 
Economically 61 43 ngs 48 ne 
Disadvantaged (3,741,755) (5,061) (5,572) 
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Figure A1. Fall-to-Winter Results: English Learners 
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Figure A2. Fall-to-Winter Results: Students with Disabilities 


Panel A: English Language Arts: Students with Disabilities versus Students without Disabilities 
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Figure A3. Fall-to-Winter Results: Race/Ethnicity 


Panel A: English Language Arts: Race/Ethnicity 
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Panel B: Math: Race/Ethnicity 
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Figure A4. Fall-to-Winter Results: Prior Achievement 


Panel A: English Language Arts: Non-Low Prior Achievement versus Low Prior Achievement 
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Figure AS. Fall-to-Winter Results: Homelessness 


Panel A: English Language Arts: Homeless versus Non-Homeless 
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Panel B: Math: Homeless versus Non-Homeless 
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Fall-to-Fall Analyses 


The sections that follow detail the analyses conducted to compare growth from fall 2019 
to fall 2020 to fall-to-fall growth In prior years. Fall-to-fall results were previously reported ina 
PACE commentary. Figure A6 depicts the data requirements for LEAs to be included in the 
fall-to-fall models. Table A2 provides the size of the fall-to-fall sample by posttest grade, growth 
year, and assessment type, and Table AS provides summary statistics for the fall-to-fall sample 
by student subgroup, year, and assessment type (and compares them to the 2019-20 California 
state sample). 


Figure A6. Local Education Agency Data Requirements for Learning Change Model: 


Fall-to-Fall Models 
At Least One Required 
/ V V COVID-19 \ 
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Table A2. Number of Students in Each Posttest Grade by Assessment and Growth Year: 
Fall-to-Fall Sample 


Panel A: MAP 


Posttest Grade 


ELA 


Fall 2017 to Fall 2018 


Math 


ELA 


Fall 2018 to Fall 2019 


Math 


ELA 


Fall 2019 to Fall 2020 


Math 


Grade 4 6,766 6,833 6,812 6,863 5,757 5,786 
Grade 5 7,301 7,371 6,795 6,855 6,103 6,157 
Grade 6 6,866 6,745 5,665 5,487 4,394 3,976 
Grade 7 6,494 6,752 5,392 5,252 4,042 4,136 
Grade 8 6,379 6,483 5,332 5,266 4,233 4,348 
Totaln 33,806 34,184 29,996 29,723 24,529 24403 


Panel B: Star 


Posttest Grade 


ELA 


Fall 2017 to Fall 2018 


Math 


ELA 


Fall 2018 to Fall 2019 


Math 


ELA 


Fall 2019 to Fall 2020 


Math 


Grade 4 4,391 1,147 4,254 1,042 3,995 1,139 
Grade 5 4,389 1,179 4,172 1,018 3,924 1,136 
Grade 6 3,913 539 4,003 498 3,637 1,058 
Grade 7 3,526 385 3,722 353 3,504 451 
Grade 8 3,627 321 3,400 330 3,410 347 
Totaln 19,646 3,571 19,551 3,241 18,270 4131 


Panel C: i-Ready 


Fall 2018 to Fall 2019 


Fall 2019 to Fall 2020 


Posttest grade ELA Math ELA Math 
Grade 4 3,448 n/a 2,968 n/a 
Grade 5 3,438 n/a 3,053 n/a 
Grade 6 3,592 n/a 3,013 n/a 
Grade 7 3,577 n/a 2,665 n/a 
Grade 8 4,195 n/a 2,980 n/a 
Total n 18,230 0 14,679 0 
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Table A3. Percent (and Number) of Students in Each Student Subgroup by Assessment and Year 
versus California State Student Population: Fall-to-Fall Sample 


Panel A: MAP 
2019-20 Fall 2017 Fall 2018 1 | 40h Re) 
California State vom g-) | 40h Re} i vom w-) | W40h he) i von z-1| 0407-40) 
Lxe) eLULE-La(oya) 
Math aN Math ELA Math 
Native American and <1 al <i <i <1 <i <i 
Pacific Islander (30,282) (92) (91) (120) (120) (108) (112) 
Asian American 9 2 2 2 1 2 2 
(575,067) (575) (577) (450) (438) (398) (396) 
Black 5 5 5 4 4 4 4 
(324,496) (1,550) (1,594) (1,261) (1,249) (876) (895) 
Latinx 55 82 83 83 83 83 83 
(3,381,198) (27,833) (28,299) (24,771) (24,683) (20,393) (20,302) 
White 22 8 8 8 8 9 8 
(1,381,737) (2,797) (2,683) (2,542) (2,402) (2,108) (2,045) 
English Learners 19 29 29 26 25 26 26 
(1,148,024) (9,772) (9,919) (7,691) (7467) (6,491) (6,312) 
Students with 12 14 14 15 15 10 10 
Disabilities (721,198) (4,836) (4,925) (4,477) (4,428) (2,519) (2,461) 
Economically 61 86 86 84 85 88 88 
Disadvantaged (3,741,755) (29,033) (29,499) (25,237) (25,173) (21,517) (21,443) 
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Panel B: Star 


2019-20 Fall 2017 Fall 2018 Fall 2019 
California State vom g-) | 40h Re} i vom w-) | 40h be) i vo g-|| 0-40-40) 
exe) oLULE-La lola) 
Math 
Native American and <1 <1 <1 <1 <1 <1 <1 
Pacific Islander (30,282) (92) (25) (99) (23) (93) (28) 
Asian American 9 7 9) 7 i) 7 6 
(575,067) (1,381) 167) (1,359) (158) (1,272 (266) 
Black 5 8 5 7 3 7 4 
(324,496) (1,548) 168) (1,463) (82) 1,278 (150) 
Latinx 55 oul SO: 52 30 oul 28 
(3,381,198) (10,032) (1,165) (10,061) 972) 9,351 (1,165) 
White 22 27 53 27 57 26 54 
(1,381,737) (5,323) (1,887) (5,250) (1,839) 4,827 (2,219) 
English Learners 19 20 Ls 21 12 21 LS 
(1,148,024) (3,916) 453) (4,018) 375) 3,915 (535) 
Students with 12 sal 15 12 14 9 11 
Disabilities (721,198) (2,205) 544) (2,296) 468) 1,563) (449) 
Economically 61 61 40 61 35 59 55 
Disadvantaged (3,741,755) (11,966) (1,441) (11,932) (1,138) (10,727) (1,435) 


Panel C: i-Ready 


2019-20 
California State 


xe) oLULE-La(ola) 


Fall 2018 
LvoW 1167-40) he) 


ELA Math 


Fall 2019 
iCoWl 1107-40740) 


ELA Ea) 


Native American and <1 <1 n/a <1 n/a 
Pacific Islander (30,282) (69) (58) 
Asian American 9 8 n/a 9 n/a 
(575,067) (1,540) (1,315) 
Black 5 4 n/a 4 n/a 
(324,496) (796) (624) 
Latinx 55 51 n/a 50 n/a 
(3,381,198) (9,290) (7,273) 
White 22 30 n/a 30 n/a 
(1,381,737) (5,420) (4,426) 
English Learners 19 13 n/a 12 n/a 
(1,148,024) (2,325) (1,755) 
Students with 12 14 n/a 9 n/a 
Disabilities (721,198) (2,526) (1,262) 
Economically 61 47 n/a 49 n/a 
Disadvantaged (3,741,755) (8,519) (7,144) 
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Fall-to-Fall Results 


Our fall-to-fall results answer the question: How much more or less did students grow 
academically from fall 2019 to fall 2020 than they did over the same time frame in prior school 
years? In other words, they measure the degree to which students grew faster or slower in the 
2019-20 school year at the initial onset of COVID-19, compared to in pre-COVID-19 school years. 


Figure A7 depicts results from the fall-to-fall learning change models across subjects, 
grades, and assessment types in units of months of learning (on the vertical axis). The results 
show a substantial learning lag in both math and ELA in nearly all cases. Using the inverse-variance 
weighted average of results across grades and assessments (which weights more heavily those 
results for a given grade-assessment Combination that are estimated more precisely), we find that, 
compared to growth in prior years, students are experiencing a learning lag of approximately 
1.1 months of learning in ELA and 14 months of learning in math. However, it is important to note 
that there are substantial differences in the learning change results across subjects, assessment 
types, and grades. Specifically, the results are not significantly different from zero in ELA for eighth 
graders on the Star and fifth and seventh graders on the i-Ready, nor in math for seventh graders 
and eighth graders on the Star. The only estimates indicating a statistically significant learning 
acceleration, rather than a learning lag, are for eighth graders in math on the MAP and for fourth 
and eighth graders in ELA on the I-Ready. 


Figure A7. Fall-to-Fall Learning Change: Overall 
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Results for Student Subgroups. We examined how our learning change results differ for 
different student subgroups by examining whether a particular subgroup (for instance, English 
learners) experienced significantly more or less learning change from fall to fall (relative to prior 
years) than their peers who were not in the subgroup. 


Figure A8 depicts the fall-to-fall results for students who were economically 
disadvantaged. The top panel displays results for ELA, and the bottom panel displays results for 
math. Blue bars indicate learning change for students who were economically disadvantaged, 
and yellow bars indicate learning change for students who were not economically disadvantaged. 
Students’ posttest grade is indicated at the top of each panel, and the assessment type is 
indicated at the bottom of each panel. Results are displayed in the months of learning metric. 
We depicted 95 percent confidence intervals as black dotted lines; these indicate that results were 
not statistically significantly different from zero if they cross the horizontal axis. It is important 

to note that the difference in learning change between the two groups may still be significantly 
different even if the dotted lines over the blue and yellow bars overlap with each other to some 
degree. If the dotted lines do not overlap at all, it is necessarily the case that learning change 
is significantly different between the two groups. For example, for the Grade 5 Star assessment 
in ELA (the second pair of bars within the Grade 5 panel), students who were economically 
disadvantaged (the blue bar) grew by approximately 2.5 months of learning less than they did | 
the past, whereas non-economically disadvantaged students grew by approximately one mont 
of learning less than they did in the past, and this difference is statistically significant (as indicated 
by the non-overlapping bars). 


J 5 


Students who were economically disadvantaged experienced, on average, 1.6 months 
of learning lag in ELA and 1.7 months of learning lag in math, whereas students who were 
non-economically disadvantaged experienced, on average, 0.1 months of learning lag in ELA 
and 0.8 months of learning lag in math. Economically disadvantaged students experienced 
significantly more learning lag than non-economically disadvantaged students in all grades and 
assessments on the MAP and Star assessments in ELA except on the Grade 5 MAP. No statistically 
Significant differences were observed in ELA on the i-Ready. The only statistically significant 
differences observed in math were on the Star in Grades 4-6. 
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Figure A8&. Fall-to-Fall Learning Change: Economic Disadvantage 
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Panel B: Math: Economically Disadvantaged versus Non-Economically Disadvantaged 
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Figure AQ depicts fall-to-fall results for English learners and non-English learners. Students 
who were English learners experienced, on average, 2.1 months of learning lag in ELA and 
1.9 months of learning lag in math, while students who were not English learners experienced, 
on average, 0.8 months of learning lag in ELA and 1.4 months of learning lag in math. 


Figure AQ. Fall-to-Fall Results: English Learners 
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Panel B: Math: English Learners versus Non-English Learners 
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Figure A10 depicts fall-to-fall results for students with disabilities and students without 
disabilities. Students with disabilities experienced, on average, 0.7 months of learning lag 
in ELA and 1.3 months of learning lag in math, while students without disabilities experienced, 
on average, 1.2 months of learning lag in ELA and 1.6 months of learning lag in math. 


Figure A10. Fall-to-Fall Results: Students with Disabilities 
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Panel B: Math: Students with Disabilities versus Students without Disabilities 
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Figure Ali depicts fall-to-fall results for students of different racial and ethnic backgrounds. 
In ELA, on average, across grades and assessment types, the learning lag experienced was 0.9 
months of learning for Native American and Pacific Islander students, 0.0 months of learning for 
Asian American students, 0.1 months of learning for Black students, and 1.7 months of learning 
for Latinx students; White students experienced 0.1 months of learning acceleration in ELA, on 
average. In math, on average, across grades and assessment types, the learning lag experienced 
was 2.0 months of learning for Native American and Pacific Islander students, 0.0 months of 
learning for Asian American students, 0.8 months of learning for Black students, 1.8 months of 
learning for Latinx students, and 0.9 months of learning for White students. 


Figure A11. Fall-to-Fall Results: Race/Ethnicity 
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Panel B: Math: Race/Ethnicity 
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Figure Al2 depicts fall-to-fall results for students with low prior achievement (defined 
as students scoring in the bottom two performance categories on each assessment??) and 
students with non-low prior achievement (defined as students scoring in the other performance 
categories on each assessment). Students with low prior achievement experienced, on average, 
14 months of learning lag in ELA and 1.6 months of learning lag in math, while students with 
non-low prior achievement experienced, on average, 0.9 months of learning lag in ELA and 
1.5 months of learning lag in math. 


10 The MAP performance categories are based on quintiles, such that students in the bottom two categories correspond to 
students in the 40th percentile or lower. The two bottom performance categories on the Star include “Did not meet standard” and 
“Nearly met standard,” and on the i-Ready include “Three or more grade levels below" and “Two grade levels below.” 
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Figure A12. Fall-to-Fall Results: Prior Achievement 


Panel A: English Language Arts: Non-Low Prior Achievement versus Low Prior Achievement 
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Panel B: Math: Non-Low Prior Achievement versus Low Prior Achievement 
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Finally, Figure A13 depicts fall-to-fall results for students experiencing homelessness and 
students not experiencing homelessness. Students experiencing homelessness experienced, on 
average, 2.1 months of learning lag in ELA and 1.9 months of learning lag in math, while students 
not experiencing homelessness experienced, on average, 1.1 months of learning lag in ELA and 


1.5 months of learning lag in math. 


Figure A113. Fall-to-Fall Results: Homelessness 


Panel A: English Language Arts: Homelessness versus 
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Panel B: Math: Homelessness versus Non-Homelessness 
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To summarize, we observed more fall-to-fall learning lag for students who were 
economically disadvantaged, English learners, and Latinx (with these results frequently statistically 
significant on the MAP and Star assessments). We also observed more fall-to-fall learning lag 
among economically disadvantaged and Latinx students in ELA on the i-Ready, although these 
results are not statistically significant in any single grade. We found mixed results for students 
with disabilities, as they experienced more learning lag on the MAP (relative to students without 
disabilities) but less learning lag on the Star in both subjects as well as on the i-Ready in ELA. 


Fall-to-Fall and Fall-to-Winter Results 


It is worth noting that we observed more average learning lag in the fall-to-winter results 
than in the fall-to-fall results. Although this suggests a continued learning lag in the 2020-21 
school year beyond the learning lag experienced in the 2019-20 school year, we caution readers 
against directly comparing the magnitude of the results from the fall-to-fall models with those 
from the fall-to-winter models. This is because these two models use a sample of students that Is 
not longitudinal (which we use in order to maximize our sample size and bring as much student 
data as we can into our analysis) and because it can be challenging to precisely measure growth 
over such a short time span (i.e., from fall 2020 to winter 2020-21). 


To probe the robustness of our results to this first issue, we restricted our fall-to-winter 
models to a matched sample of students also included in the fall-to-fall models. We found that, 
overall, the differences between fall-to-fall and fall-to-winter learning change were robust to 
this restriction in models with the MAP in ELA, the MAP in math, and the Star in ELA, suggesting 
that the difference in fall-to-fall and fall-to-winter learning change was not due to differences 
in the fall-to-fall and fall-to-winter samples used for these assessments and subjects. In contrast, 
we found that the difference between fall-to-fall and fall-to-winter learning change was more 
sensitive to this restriction in the smaller samples used for the Star in math and for the i-Ready; 
that stated, the broad result of measuring more learning lag over the fall-to-winter period in 
comparison to the fall-to-fall period held both with and without the sample restriction. Although 
we feel this is further evidence of continued learning lag this school year, we still caution against 
a direct comparison of the magnitudes of the results from each model, given that it is still difficult 
to measure and compare growth over the short same-year fall-to-winter period. 
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Improving education policy and practice and advancing equity 
through evidence 


PACE Is an independent, non-partisan research center led by faculty directors at 
Stanford University, the University of Southern California, the University of California 
Davis, the University of California Los Angeles, and the University of California 
Berkeley. Founded in 1983, PACE bridges the gap between research, policy, and 
practice, working with scholars from California’s leading universities and with 

state and local decision makers to achieve improvement in performance and 
more equitable outcomes at all levels of California's education system, from early 
childhood to postsecondary education and training. We do this through: 


ch bringing evidence to bear on the most critical issues facing our state; 
€ making research evidence accessible; and 
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